DOCQHEHT BESUHE 



BD 113 376 

■ • ■ - v .( 

iUTHOB' . V 
TITLE 

PUB DATE 
/ NOTE 



EDBS /PBICE , 
DESGlilPTOES 



TM 004 873 



IDENTIFIEBS 



HoonaDr *f)illian J. 
Contributions to Objective Measuremeni: and Evaluation 
of Trainee >(Corapetency» 

Oct' 74" X , , 

33p,; Y^per presented at the Coiiference of the^ 
Military Testiftg Association (Oklahoma City^ . 
Oklahoma^ Oct. ^,1974) 



MF~$0.76 HC--$i;95^Plus Postage 
CQiopater Programs; Goodness of Fit; Item Analysis; 
♦Hathematical Models; Matrices; *Mea'surement 
Techniques; *Military Personnel; *Military Training; 
♦Personnel Evaluation; Probability; Statistical 
Analysis; Systems Development ; Trainees 
Binary '.Measurement M^del; *COMET; Frequency Ratio 
' Method ^ 



^ABSTEACT ^. ' . 

The purpose of this paper is to lay a basis for and, 
discuss the components of a system^ called COMETr designed to 
• objectively measure and evaluate the competency of tifainees in - 
. military ^train^ng enterprises^ COMET is an acronym for "Computerized . 
Objective Measurement and Evaluation of /Trainees* " These goals |will 
be accomplished by: (a) describing what th^ author means by the^j terms 
"measurement^" "objectivity r"' and "evaiuationv " by indicating sVgtems 
antecedent to COMET r and describing the objective Biftary Measurement 
Model (BMH) used in the GOMET system/ (b) deriving and illjastrating a 
new procedure for . estimating one of the parameters, and fit of t)^ BMM 
modelV called the Frequency Eatio 'Method (FBM) r' and (c) specifying > 
the compo'nent subisystemsjr and their functions^ of . COMET as well as 
its goal objectives aiid practical problems associated with its : : 
implementation. A detailed numerical example^iis provided td ^ 
facilitate understanding of COMET* s mathematical ojiA statistical 
aspects. (Author) , * ^ 
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- 1. Introductio'h : . * " ^ • ' t 

/ The purpose of this papex is to lay a basils fpr and discuss the com- 

" . , ■ ■ ' ■ > />■ ■ ■ - •■■ - ■ ■ ^ 

y'ponents of . a system, called COMET, designed ^^^to^ob'jectively measure 'aiid 

^ 'evaluate' thie compe,tency of trainees in military training .enterprise^. . 

^ . . • - ' 1* ' ' • • . ■ ' ' ^ 
COMET is an acronym for "Computerized Obi^ective-Meiasureinent ^hd Eval^r 

. / '• • . • ■ . ■ . * . * . . 

) ■ » ♦ . ' ■ . ■ ■ ' ■ * ti ■ . , 

uati^n of Trainees." 
. y These, goals will be acconjplishfed by: ' '\ 

V / ' ■ ' ' • ■ ' - m ' ^ ■ / / ' 

/ a) .describing what the author mean^by tW tempiS "measurement 

• A . • -jv ' r ■ ^ ^ • /i ■ ^ / , 

/^Si^ • "objectivity,"; and "evaluation," by indicating systems antecedent 
f to COMET,' and describing the oi([jective Binary Measurement Model 

^ '(BMM) used in the COMET system,* / 1 y 

\ . . • • ■ ' /. • ■ 

b) deriving and; illustrating a new .ptpcedure for estimating ^oij^e of ' 
the. parameters arid fit pf the BMM model, called the Frequency • 

| Ratio Method (?RM) , and . /' 

c) - specifying the component subsystems,; and their functions, of COMET 

as well as its. goal, objectives and pyacj^trLcal problems associated 
\ V ^ with its implementatior>. ' / Xr^ 

A detailed numerical example is provid^ to. facilitate understanding 

■ \ I ■ " ■ . . ■ . / ■ ■ ■ .- . . 

of COMET'S mathematical and statis,tical aspects. ^ - 



2. Background 

By the term "measurement" the? authc(r refers to the process by which 



numbe^ are assigned vto a property ofy^an entity. I^i training enter- 
prises, the entities ^re personnel oi: trainees whereas the property ; 
being measured we refer to ^s' "cbiyipietency" in the tf aining, f ielH^ 

By the temf "evaluation" the- author) refers to, the process by which 

■ ■ ^''/ ^ ■ - r - 

mp;l.surements or the'xr functie^s are used to characterize features or ^ 

^ ./ V '/ ■ . ' ■ . ■ ' 'r'" 

aK,>ects of thj^valuated "situation", or the entities included theirein. 



For "example, we may evaluate- a trainee (entity) during t^ain^ng ' 
(Sittiation) by -thfe pi^ocess of taking the difference ^unctiofi) of his 
■ -.'competency (measurement) as determined . at two different stagesi^of train- 
••••ing.. NoteLce that ^ other value "judgment Is placfd on the evaluation 
result: in this definition. This w^s omitted -becaute the value applied 
^o^an eValuatUon result is not uniqtie. It ,±s« different depending upon, 
■ the decisibri maker atld the, decision problem wherein the.°evaluatibtiX^re^. 
' • . . 1:0. be Utilized for. decision purposes.'' If the results are not to be • 
.^utilized for d^Qision purposes,^ they need no value judgment component, . 
^ .By the tW '.'ob jectiveV the' author means <ii^^ depending 
upon what is reterred "to. . There are. ^hree different senses in^ which the 
.:" . term Is'fcased in this paper. They are ' 



• a) training program objec-tivity, . ^ . "o. 

b). Measurement Instrument pbfjectivity» 4^ 
• •■ c) -Measurement model parameter object iyiiyi / / . 

' ' Training Program Objectivity refers to^the process wher4y training 
programs are objectively established by meand of expressly st^ed golls, 
functions and objectives and are designed* and managed by a carefully , 
thought out, systems applroach such as that -developed ^y Rundquist (1967 ) . 

Measurement In strum ent Ob.jecttvity refers to prbcess .whereby, a m^'a- 
sur^ment instrument (test) .is desig-ned to be flee of ambiguity and sub- 
jectivity in its administration or- in respfendi^l^g to or scoring its items. 
. Measurement Model pkrameter -Objectivity reSers to 'tWo. propetties of ,^ 

-a measuremerLt^mpdelj viZ' . l-' ' ' \, / 

a) Che comparison of any two tiestees may ;We carried' out in s^^ a, / ; 



r 



/ 



way ttJ^t no other measurement parameters are in^^olved than those of .j/he ^: 
two testees, and ^ 

■ < • ' * 3^7 



, b) any two items may, be comparisd independently of all others ' ^ 

param^j^ers '(including those of the testees) than. those of the "two items. 

■w| 'emphasise the desirability of requiring an evaluation system, 

automated or/not, to possess the three aspects of ^objectivity, otherwise 

it is likely not .to meet its goal and objective&^as well as not pirovid- 

Ixig^^eiiil measAire&ents and ^evaluations, * 

A0 f ar .as is known, there is only one. class of measurement model 
' .' " • . ' ■ ■ ■ •' 

(thos'e developed by Rasch" (1960) which possess dhe properties of mea- 

s4rement models objectivity. That th^se properties actually exist for 

some of the'R^sch models, has been shown by Rasch him?elf (1969), Wright 

(1969), Choppin (1968), and Schmidt (1969). Later in this paper we will 

provide ye q, another:'' demonstration^ A incest, startling and readable 

account of „the Rksch M\s objective' features has been provided hy 

' ^ ■ -. . J • * - ■ ■ ■' • 

%right?> (1968) . My own accoDant , Moonan U9Z2) ,• may be of use and con- 

• ■■ "■ ^ <■■ : ■ 

■ . - ^ ■ - ■ \ : 

veniently available to you. • 

^ 'We are aware of- at least ^wd antecedent, automated trainee evalua- 
txoii systems, ^ere'are ^>robably otliers, but' our attehtion has not^en 
directed' to them. The designers deserve our applause for their pioneeif- 
' ing worl<C and we can.onl^ h±s\ that the COMET system we pro'pose could be 
-implemented as effectively and •successfully as eachof^these were. The 

first system, I refer to'W"(?LAKES,-" was developed by CDU W. H. Wheeler 

. . ' • ■ ' <■ ■ " " . . . . ' . ■ .' ■ ■ ' -f 

wHiie stationed ^at the Navy Electronic Technician's School at the Naval 

Training Center, Great Lakes, jriinois. My documentation. Wheeler ^1966) 

is meager, but I did s.ee ^t in operation. I was partrcularly impressed 

- ' . '■ ■ ■ ■ . ■ , ' 

hy its te^st reprodu^itipn^subsystem wherein examinations, were quickly and 

c'onveftiently assembled, from 'tli'e 'item .files. Its evaluation subsystem . 

\ • ■ . ^ ■ „ ■ : ■ ~. i ' ^-^ - 

was also excellently designed, * ' ^ , , . ^ 
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\ .. The second .system x^as develoj^ed for the Marine. .Corps Communicatioha 
ElQctronics School at th« Marine Corps Kqcruit' Depot, San Di^go by Dr. 
Richard S. Hatch, (Decisions Systems Associates, Inc. , Rockvillfe, ^) • 
This system is known by the acronym ATAC;» Automatic Testing and^ Attrition 
Control and, as its natne implies, 'contained a subsystem for controlling^ . 
how many, which,- and x^hen trainees' should be attrited from the training ^, 
program/ This' 'system had other excellent features as well. Mpst parti-7 
eular were its reporting subsystem and th^ evaluation subsystem. 

• Both systems were ahead of their jtime, the BMM was not well kno^^Tn 
nor computationally effective the time of "their design. Theij develop 
ment shows that'.only "bare bones" d^^sign features .ca^, .at . first , be imple 

men ted /but must ultimately. be augmented by features which fulfill local ' 

' t. - '•■'^ - ■ ■' . 3 , .■ ' ^ ■ \ ^ • ' ■ • '. 

requirements, overcome constraints ariid meet Ipcal desires.- • It is 

believed that the basic COMET system,' described heteiri,. supt>lies the . 

essential /*lfare bones" for many imp lemenlat ion situations, - • 

Ifc should'be fiientioned here, ^most'^'jEittingly-, that^my own particular 

interest in this research area w^s substantially motivated by seeing 

Gl^^KES^in operation, working -with Dr^ Itti^tch on ATAC and Vbe^ 

by CAPT 0. L. Dawson, USCG, to develop in effective COMET-like system for 

the U.S. Coast Guard Training Facility in New York. r iV;, ' 

My personal documentation on ATA is ^imited, ' Bl^kely (19^9) , aitd 

'hatch (1967) but, perhaps Dr; Hatch could supply additional information 

- ' ■ .:„: : ' 

if it^is needed. 



3» The Binary Measurement Model ' ^ 

. Rasch (1960) developed, essentially,, three sub-classe« of a general 
class of objective meksurement models. 1 refer t</ these models as CD, 
AD and SP models as 'described ■in_ Exhibit 1. Actually the AD model is , 
sufficiently general tp.encomp'ass the CD model which in -turn can be • " 
shown to mathematicaH5? encompass SP models." The BMM is the AD model 
and is the one dis1:ussed previ^ously (Mbonan, ^1972) . The mathematical 
statement of the model is expressed In probability terms.- Th6 proba- 
bility of a response,. X(i,j)" being- correct, X^.^^^ = 1, when made to the 
ith it6m,' witfe easiuess,. e(i), by the txth trainee who has- competericy 
Y(n) is given' by the expression, where j^^n, : \ ' . ' 

< a) 'PCX.. .,=1 I e.(i).^ Y(j)]'= ^^^^ "' "^^^ = P, 

■ V I ^ M-J-'-l l+ed) • Y(n) * 

.-■ • " ' ■■■■ ■ ' "^'V ' ■ ' ■ ■ 

Similarly, the proba'bility 'of an incorirect r^esponse is given by * 



The m^^^ L 
allowed to be binary or dichotomous'. "Responses to Affective Domain 
items are usually, n-ary so that the Rasch polychot6mous model is" more 
appropriate for measurement under .such •conditions . In y/his case, when. 
n=2\ the GD:BMM^model is equivalent to the. AD sub-model which we will not 
discuss further ati this time. • ) 

' The model produces >easuremen^^ because its use permits the assign- 
ment of numbers Co th6 ''cpmpet6hcyM. of trainees. It also "measures" the 
Items since the easiness ^rC^ of each, item .may b„e numerically esfeimated. 



The scale of competency has a zero point and an estimable unit of Tnea- 

-■ ' • . • / ' ■ ■ ■ ( 

suremeht; Thus, the model, with its objective properties proyides a 

I. . ■ • .• ,• " . ' ■ ' . ' ^ 

suitable measurement basi& for, the COMET system, . / * _ ^ 

* •. . ' • 

The BMMhas two parameters, Y(j)"and E(i). The first -we call "com- 
petency" "rather than the more usual "ability." ' We prefer, the former 
term for draining situations and have avoided the confusion sometimes " 
resulting from the word "ability'." > The 'easiness parameter estimated • ' 
indicates how ."easy" or -"difficult-" an-item's correct response is to 
discerh f o^, a tgstee. , • 

4. Model ' Ob.1 ectlvity . > ' ; , ' 

As'-'was pointed out in. earlier sections , Objectivity plays an impor- 
tant role in fhe COMETTsj^st^m. In this section we shall examine the 
model objdttivity of the BMM by , considering the'dualistic concepts and 

implications of measurement model objectivity. 'These- duals are listed^ 

- ■ ■ ^' . • . ° , » • ■ ■ ■ 

below: ' ^ • , ' * 

. Concept No. 1 . • The'^item pardmet^ers , E(i), caa be estimated inde- ' 
pendencly and without knowledge of °the competency 'p^raKeters, y/n) . of 
uhe persons responding to the itera,§. 

Implication NO. 1 ." The* easiness .scale can be\calibTated and an 
easiness can be estimated independently, of th^ persons , -and their com- 
petencies, responding to the items. This implication is" very impQ^rtant 
for the COMET system for it means we ca^ collect item r^sijonses data 
from whomever it p.s convenient "to do so.^ Items can be administered 'in 
small sets of items to persons oi quite divergent competencies. -Groups 
of similar persons are not required^ to respond to the same items. 



' ^Concept No, 2. The competency parameters, yCn) , can* be estimated 

' independently and without, knowledge of 'ttie item paramet(ar§,y £/('!)• 

'• ■ ' ~ \^ ( . ' " ' * • • ■ . - ... ^ / ■ ^ 

■ ' Implicatiorn No , 2- , ' The competency scale cap be calibrated and a^ 
• competency can b^ measured inde^pendentl^of the items used for measure 
ment/ Simply stated this means tliat, if desired* individualized tests 
for 6ach trainee could be used, but^. in doing so, ' 6ach trainee is mea- 

• • , ■ ■ . ' . ■ ^ ' • ■ 

. sifred on .the same competency scale • » • 

■ ■ ■ • • " ■ • o' ■ * ' ■ " 

/ ^ we shall inathemat.ically prove the two concept statem^ts ■ for ^ the 

. BMM below: , / . *. • . • ' ^ ^ 

■ * ■ , " • ■ " • • • " * ■ - . '- 

" ^ According to the/ model we have the single item probabilities ' . 



(3) 



(4); P[X(i.n)= 0] = . 



X 




, Consider next the probabilities associated jtfith the four pdss^ble^ 

person, nr^ responds to two items, ^ 

i /atiS,^:;^': /To this ^-end .^let,,.^..^;;. i ....... ' ' ■ ' > 

(5)* p(i) =1'+ e(i)'Y(n) and . D(j) = 1 + '-£(3) Y(n) Jje the 
'^'^flenominator' terms of x)ur ^probability expressions, then 
(assuming response indepetitdenc^) ^ ; , 



•(6); PfX(i,n) = 0, XCj.n) = Oj;- p(i)^j3(j) 



<8') 





•■.x>(j) 




• Y(n) 




• D(j) 




• Y(n) 



■ . ■. • . - .^ . ■ .■ • V 

Vq\ prY^n^ - 1 ■ yfn 'n\ - n e(x) • Y(n)' - e(.i) • Y(n) ^ 
(9) -P[X(i,n) - 1, X(3,n) - 1] ..- - 't . D(j) " ; 

■ . . • • * ' ■ ', 

V TKe conditicThal probabilitj^ ,that tlje nth person gets the 'ith item * / 

. correct, give'n that he scored a total of 1 on both item i and j is€?given 

. ■ - > ■ . , ' ■ - : - ' ' : : ■■ Q ■ 

by the equation raCio (11) = (8)/(lO) whete (lO) is given by the., equa-' _ 
. tion sum (10) = (7) +' (8). Thus - °- ■ % . 

■ ' ■ ■ . , ■ . " ■ ■ ' _ •; ■. " ■ 

^(11) " = ' /.f^t^ /.N ;in which we note the Y(n)° cancels," 

• (10) £:(i) + e(j) '"^^ , . . 

and which indicates the truth of ^Concept No, ^1, for suppose we haveVg|ol- 

• ■ . • . ' » ' ■ ■ . 

Icected responses made 15^ a set o'f persons to itenis i and j, and suppose 

Phat ■ , ^ . \ ■ i--- 



02) f(i^i) of them got -item.i correct and item j incorr.ect^ and^ ^ 
(13) f(3 ,k) of them' got item ^j^ carrfect and itfeip i incorrect, we note 



by i ' - . 

. • . • ' 

* -, (14) g(i,j) f (i,j) + f(j»i) ^I^e tlumbj^rVGf these persons who 

» receive a total score of^l on both of the. items. 

^ ■ ■ » , \^ ' ' ^- . ■ . 

^ ■ . (t. . , ■ 

' * We notej additiohally,"-feha^ ' 



(15) ^^4^ estimates '-^7^ 



Ci) 



> ^ ■ ■ / ^ . • * ' * ^ 

(16)^^iA-i)^-^^^.i"'^^^"^ E^i) i e(j). ' ^"'^ '7 • 

t ' ■ ■ 

(17;.^7f4r estimates >' , f ur'thet substantiating Cdncept No. 1 

/ f (j ,1) •6:(j ) . ' y \ 
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. ♦ ■ • . ■ » . , • .. ^ - .v^-' 

'**-•* • ^ ' • •• - • 

Eqn (17) was derived by Rasch (1970) , Choppin (1968) / and others, : j ' 
but none, to my knoWl^idge, have used this fact to develop ari ^(i) estx-^ . 
mation method, based on the* frequency ratios.*^ This proces^ kifown as tlie 

*' " * •' ' V'i / " " . * . '• V '■ 

Frequency Ratio Me&hod»^4]^M represents thV author ' s modest "contri- ; v 
bution" to bbj active .measuir^ent? and forms a computationa^l basis *for the 
COMET system.. Exhibit 2 shows some available item easitt^ss computational 

. methods. . ' • . ..' ; • 

To continue our derivations consider 'an -i-tem, >i, responded to by two ' 
persons with* different competencies ^ y(m) ana . y (n) . .A^ain there are 4 

. outcomes and ° their probabilities are listed bel^jg/: • * . 

: . ■ ' . .;. . ,. ■r'-'a-. r ■■• ■ • 

. ,,, . :(18) ^'^et" iD(m) = l; +. eCl) y(.m) and D(n) = ,1 eCi) '"^ T 
\, \ denominator terms in the' probability expressions 





_ PlX(ma) = 


■0, 


jC(ri,i) 


= 0] 


^ :(20)i, 


P[X(m,iV - 


h 


X(n,i) 


= 0] 


(21) 


PlXXm^i) = 


0, 


X(n,3^ 




(22) 


P[X(m,i) = 


^' 


X(n,l) 


= 1] 













D(m) • 


P(n) / 


' . .. » 


e(i) • 




I, 


l>(?i) • 


•D(n) 




e(i) • 


y(n) °: . 




D(m> • 


P(n)' * 






• Y(m) • Y(n) 





P(m) • D(n) 



The (Conditional probability^ that the ith item was correctly responded 
\ j to by the mth person given that the total score made , by the mth and nith 
persons to that item was equal to 1 is given by ''the equation ratio 
(24) = (20)7(23) where .(23) is givert by the equation sum (23)-(20)+(21) 
thus • ' ' . ' • . 



- / 



(23) P[X(m,i) + X(n,i) = IJ 



g(l) • {Y(m) + Y(n)} 
D(m) • D(n) 



» and 
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(24). • ' 



(-20) 



Y(in) 



(23) . Y(m)'+ Y(n)^ 



which indicates the -truth of- Concept No* 2, for .Suppose x^e have collect/ed 

respoti^es made to a .set of items by pfersojis in aiiidi u and suppose ^ 

>. . * ' " / : ' • * •1 ; ■ ' . 

(25) F(ii^,n) of the items were correctly .tedfipnded to by. person m 

, ai^d were incorrectly responded to by:,4)ersort n, and ^ 

(26) F(n,m) of ^the i^ems were correptj^iy responded to^ by person n 



• ;and were incorrectl^r^ responded tdfl bj^i^^rson m, We also let 



(27)' G(m,n) = F(m,n) 4- F(n,m) be the :xiiumbet of items which werg^ 
responded to cprrectly by person^^m t>r n, ^ ^ ^ 



\le' can see immediately that 



(28). 



(29) 



G(in,n) . 

F(h,.mK : • 
G(m,n) 



estimates'* 



estimates 



Y(m> ; ' 



(30) li^'^l estimates 
F(n,m) 



Y(m) +i'^n) 

Y(n> 
Y(m) + Y(n) 

Y(in) 
Y(n) 



and . 



, whereas 



further subs tan- 



. tiating Concept: No* 2. , " . * 

- ■ ■ ■ ■ . ^ ■ ., ' • 

Rasch (1970) indicates that (30) cannot be practically used to esti- 

.*>'*■ 

mite y(n) siace^he numt^er of it%ms used iii an assessment program is 

. ..^ , ..... . ^ ^ 

usually .much smaller than the numb^^r of testees. Consequently we will 
use atiotlier * approach to estimate Y(n) in the FRM or the measurement 

model analysis sub-*system of : COMET, although, it seems clear, that iV 

■ ' V " . ' ■ .r ■ ,■ ■ -i":/.^ ■ ■ . • . ' , . 

some ass§ssment situations, the nunibet of items ^s sufficient if the* 



< 
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number of yCn) to be estimated corresponds to the number of possible raw. 

scores. * * 

* ■'■' "■•]? ■ ' ' 

5. The Frequendy Ratio Method (FRM) v / ' ' ^ 

.The FRM is a^athematical-computational procedure designed to provide 

a simple estimation and goodness of ^ fit test for the BMM parameters and 

i ' " ■ * 

responses. It is intende^ to be an analytical component of v the oCOMET 

system* To illustrate FRM ve assume we have collected from trainee- 

persons their rfespon^es to the available litems of an iteni bank ^nstruoted^ 

from the learning (training) objectjiv^s'^of the training course and % * 

assumed td indicate training competency if the items'^are correctly 

responded to by the trainees • ' Our available data (S^n be portrayed as the 

frequencies- of equations (12) and (d^) and arrayed inva square matrix 

R(i,j) wHose main diagonal is left blank. . - . 

After portraying. the mathematics and statistics of the FRM we will 

■ ■ ' • ■ .. • . . ■ ..■ . .... -.^ ■ • _ 

formulate* the Computations in two stepwise phases. ^ . ' 

Phas6 I: Item easiness estimation computations. ^' 
PhaVe II: Goodness of Fit computations. • 
,In R(l,j^ we form ratios of symmetrical elements and form the squ^e 
matrix S(i,j) . ' Thus ^ ^ 

' ' ' • ♦ ' 

(31) " R(i.j)' = |7Hf ^=^"1' V' i» ' • 

(32.) Rd»i) = f(i;>jj, for j>i, j=iFX, .... I, and . • 

If . . ■ ■ . ■ . ■ 

^3) ' R(i,j) = blanic for i=l, .... I. 

We know from (1^) that (31) estimates e(i)/e(j) and (32) estimat-es f 
e(j)/e(i) so that the" natural logarithm of (31) is ^ 

■ ■ 406 * ■ • 



(34) , In S(i,j) = In e(i) - In e(j) whereas for (32) we have 



' (35f In Sri.i) = In e(j) - In^ed) = -In S(i,3) ' ' /. ' ' 

If v/e sum the jt'" rox^ of- In S(i,j) we get, after adding "and subtrJjkcting €(j), 

(36) T(j) = Eln'eU) - I In £( 



' and^ let 



i=l 




(37) Eln e(i) = 0 for def initeness; and 

In eg) = ^^|§7 =' t(3)-, and 3 = 1, . . X, so that 

r 

fc(3) = e . » 



A stepwise 9oinputational proeed^fe^is l-ndicated xn^ Exhibit 3- Th^s • ^ 

' simple estimation, procedure orequires only£tie development and. mainten- 

. ' ence of. a* square matrix of-f jrequenc'ies whose elements are the numbers or 

y . . those persons 'tested on some of^ the items and whp responded correctly to 

one, and only one,- each pai'r of items presented to 'them. A list of 

-' . ■ ■ ' ' ^ 

item easiness eg'timation procedures are shown in Exhibit 2. 

Eqn (39) is the culmination of the item easiness estimation phase Of 
the FRM: "we now proceefl to develop the second phase which is intended 
. to provide a goodness of fit test of the -response data to the BMM.^^^^ • 
With (39) 'we are provided with estimates, ^(i), of xhe easiness para- 
meters, e(i). We intend to u& the distribution (Fisher (1948), as 
the basis of our goodness of fit test.. The hypothesis tested is that ^ 

the it"ems"*-^ed for' assessflient of /competency evoke responses in accordance 

. - • \ ' ■ ■ . ■ - 

with the BMM. The chi-square fotm we use is / ; / 
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using (12) and (13) for^the f (djf) observed ficequencies. and substantiat- 
ing (39) in (15) we define the expected frequeitbdes as^ 

' e(i') ^ . V 

<^1) e(i,j) = g(i,j) • g^^j , i>a, and. «^ 



(42) e(j,i) = g(i^-) ^j!,:. . j>i 

« „ e(i) + e(j) 



We can (^mp'ut.e contributions to the upper and lower parts wifh 



Mrhe follomng formalae 



, (43)" x^C-ivj) f [f(i,i) - e(i,j)]2/e(i,j), i:^j;'and 
(44). ,x2(j,i) = [f(j,i) -'e(jVi)]2/e(q,i), j>i^ . . " 



We next add these values synnnetrical^.y into a single triangular 



, array by the formula 



_^(45) ^ C(i,j) = x^(i,j) -^'.X^(i>i) , ; 'V 

. .considered as an array with a total of 1(1-2) degrees of "freedom, so 

/" . ■ ^ . ■ ■ - ' ' ■ ■.■ • 

that t;;ere we to cpiSpute a x^(k) for each item from tkis table, it would 

' have 1-12 degr^l^^s of freedom. We compute x^(fc) frb?i G(i,j) by summing *. 
• , . " <^ . ' " 

*=^elements,,in an "L-shaped" fatijj^on acctsrding to. the following formula 

k-1 I 

(46) ^ jcHk) - I e(f,k) + z:. ^c(k,j),^jc-i, 1 . - 



a<j.l,ectedAu level, the hypothesis that t% refsponses Lo Llie - Item f i U^^^ 
BMM is rejected and th-e/ item should be r^ove4 f rom^ the COMKT HytJtem, 



4 



V 



Exhibit 4 shows the stepvrf.se computations for these FRM. (phase II) ^ 

;'' , • c. , ■ ' ■ . ^ 

goodness of fit calculation^. • ^ ' ' * " / 

/ /■' •■■ 

. 6. A Nu merical Example (ghase 1) y . 

This illustration i^^intended to show the numerical^aspe^ts , result- ; 
ing, from applying "the FRMtQ a set ^ response 4ata. The (lata origi- 
nated' from. the responses ^of Na^ral Recruits to the first ^=7 items of v 
■ theykectrdfiic Teeh/ician Selection Test (ETST) . Orig:^Aally 9 items , , 
x^ere ased t^pt Ghe/sth and ^th itfem responses were, f oijid not to fit the 
BMM model so they were omitted from the analysis. Ml recruits responded 
to the same i items so the. data does not result frOa^the "catch as catcH 
/ ^can;* response saniplin^ procedure mentioned ear\ie,t. This difference 



causes no/problem in bur analysis. 



the k (i , j ) matrix^ of frequencies 
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ITEM INCORRECT 
»«i ■ 
4 5 



-I 



220 210 97 203 '.202 269" 
>; 157. 87 , ,_163-. 148 198. 
160 - . ' 62 ' 164 • "205 



■183 275 248 V ^ 277 

85 147 1^5^ 73 - 

164 212 . 208 U23 210 ^ - 

413 144 ' 14« >88 164 127 



2|i7 33a 
130 202 



245 



jt3 
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^ Compute Freq(uency Ratios R(i,j) 

1 2 • ' 3 4 



1 




1.864 


1.892 


.530 


2.388 


1.232,. 


2.381 


2 


.536 




.981 


.316 


1.109 

1 


i -.698 


1.375 


3 


.529 


1.019 


s 


.250 


' 1.131 


. .707 


1.385 


. 4 


,.887 


3.161 


4.000 




3.795 


2.008' 


3.750 • 


' 5 


.419 


,.902 


. . 884 


• . 264 




.619 


1.232 


6 


. 812 


1.432 


1'.415 


.498 


1.615 




i:929 ^, 


7 


: .420 


.727 


.722 


.2.67' 


,812 


..^5l8 




V 

0 
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° In R(i,j) 







1 ' 


2 1 


•l3 ■ 


■ 4 


5 \. 6': 


7 


i 


. j. 


1 . • 

- . ■ 


.622 


.638* 

* 


. -.635 


;,.'87jd . .208 


.867 


2 


■ r 


.622 


— 


-.019 


-1.151 


.103 » \359 


.318 


3, 


L 


.,638 


.019 




° 1.386 


.123 -.347 


.326 


4 


i 


.635 


1.1^1 


1.386 




11. 3*31 .69*7 


i.321 


v5 

\ 


1 ' 


;870'' 


•-.103 


-.123 


-1.331 


- -.480 


.208 


\ 

6 


I 


.208 


.359 


^.347 


- -.697 


.480' - 


.657 


7 


■1 


.867 ' 


-.318 


-^326 


-1.321 


-.208 -.657 

1 • 





ComWe T(jy, t(i) and E(j) 







■#- 


e(.i) 


1 


.2.570 , 


y-367 , 


.693 


2 


-1.730^^ 




r. 280 


3 


>1.903 


^2>2 


,1.313 


4 


" 6.521 


-.932 


.394 


5 


~2,699. 


.386 


1.471 


6 


.438 


-.134. 


.874 


7 


1-3. 1697 


.528 


1*696 



-This completes Phase 



Phase. 21" Goodness of Fit Test ^ 

— - — ^ r 



the e(i) values ^re givpn above in Step 6 of Phase^l , 



the S(ij) matrix, containing If (i, j) and f (j ,i) , is in 



Sted 1 of Phase (1). The g(i,j) matrix is given below 





1 2 3 


4 


' , 5 


6 


7 


1 


338 321 


280 


288 


366 


782 


2 


. ~ • 317. 


362 


310 


360 


342 


3 




310 


309 


' 355 


3^ 


4 






350 


370 


418 


,5 


(Symmetrical) 






340 


366 


6 










372 


7- 














(/ . * ■ 


410 














■ i 



















Expected Frequencies 

1 2 3. 



1 
2 
3 
4 
5 



219.29 210.09 101.50 

160.53 85.21 

' 110.91 156.47 — 71.55 

178.50 276.79 238._45 -- 

92.23 143.31 .145.72 73.94 

1:5^1.85 213.91 213.14 114.95 

"lip. 82 147.09 154.05 78.79 

P- Contributions - 



195'. 78' 2f04.15 
165.76 14 6. .09 
163.28 i41.86- 
276/ 05 255.04 
, ,,126.72 
^13X28 , ~. ■. 
I7O.I0I 12^.52 



271.18 
194.91 
198,95 
339.^21 
195.99 
245.48 





1 


2 


3 4 


5'i 




.7 


1 ' 






vOOO .200 


"""^ 
.266 


.023 


.,018 


2 


^ .064 




.07.8 .038 


.o4 




.049 








1> 275 


.003 


• 3186 


1 .184 




. 113 


M12 


.3?2 ; " — . 


.003 


' .253 


.250 


5 


'."565 


.053 


.004 .012 




.085. 




6 


■.0f9 • 


.017 * 


.124 . ; .562 


S050 




.600 


' 7 
Accum 


.043 / 


. 065 " . 


;238. A. 077 


.•.212 


.002 


> 


ulated X 


^'values 






1 ■ 


< 


■ 1-" ■ ■ • 


2 


3 4 


5 


"6^ ■ 


7 




.006 " 


.000 .313 


.831° 


.052 


,061 


2 ' 






.158 .050 


.099 


' .04? 


.114 


.3- 






— • 1.657 


.007 


.03i 


.422 


4. 








.015 


' .813 • 


1..327 


5 










.b02 


1.077 


6 ' . 






V . ' 






2.322 


7 

a = . 


05, df-*,- 


1-2 = 5 

"V- 


•CX2(^95, 5) = 


11.07.0 


°- 


















1 




' 411 









6 • 
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I 



Noy?(k) exceeds or equals therefore Hq is accepted 
all \ ;^teins. 

no items are removed, it is not necessary* to ^e- 




This completes Phase -2. 
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. ' EXHIBIT 1 

$p)es of Measurement Models Developed by Rasch (1960) 



Tx£e 
CD 



AD, 



SP 



Name 



Cognitive Doma:2h 



Af f '^c tive^ ^Domain 



Stochastic Process 



Application Fields ■ 

Measurement of Intelligence, Apti-- 
tudeSji Competencies .and Abilities 

Measiirement of Attitudes, Values ^ 
Motivation and Interests ; 

Measurement involving stochastic 
prbjiiess obi^'ervatio^s such as ,| 
reading errors 



exhibit' 2 > 
• Item Eai^/iness Estimation Procedures 



Method Name 
Least Squaresv \^ 
Gra{Jhic 

Maximum ^Likelihood 
Logarithmic 
Frequency Ratio 



■/ 



Oyii,Klnator 
Brafllble (1969) ■ 
Rasch .(I960) 

Wright , and Panchapakesan (1969) 
Blpmmers (1965) . 
Moonan (1974) 
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• * ^ ^ * fiXHIBIT 3 

Stepwise Procedure fox FBM' (Phase I) ' 
Item Easiness' Estimation Computations 

. ^'Collect responses ^1 the items in the item competency bank. 



1 Organize the responses into.a square matrix S(i,j) whose ^ 

elements are defined by (12) , (13) and (14) . ^ 

2 Cdmpute the frequen,cy ratios as indicated by (31) and' (32X. 

Call the matrix" o£ ratios R(i,j), * )> 

3 Take the natural Ibgaf ithms of: the R(i,j) elements; called.^ 

T(i,j). ; ' . 4 ; 

4/^ Sum each row of . the:jnat|rik of logarithms and store in a 
^ vector T(j)'- ' / . / ^ 



/ 



5 Compute t(j) = -T(j)/(3;-l) 

6 , Determine the easiness estidLmates by the relation e(o) e 



• r 



EXHIB]^ 4 



'Stepwise Procedure > for FKM (Phase II) 
. Goodness of Tit Computation 



Ste£ 



■ - .. _ . ■ • . : - . . \ . , . ■ • . 

0 Givetx. tj;ie easiness estimates 2(i) of the items whose response 

fit is? desired to be tested, - ' 

1 Specify the f(i,j)^bserved frequencies as def ine4 by eqris ""(12) 

and (13) far those items as well as' g(i,j) of eqn\ (14). 

.2/ Gompute the expire ted frequencies as given by (41) and (42).- 

3 Compute the ; contribution as defined by (43) anji (44)*. 

4 Calculate the accumulated contributions of (45). 

5 Select an yct level iS^r the significance test and find x (l""ot>^^2)=Xo* 

' ' \ - ^ ' \ ' ■ V 

6 Ccnapute the test statistics, x 00 > according to (46) , 

•7 If x^(l^)^Xo any k, reject Hq- for those items and remove them 

from the analysis.^ ' ' 

. ■ • . • ■ ■ ^ ' i • _ . , ' 

8 If any item is removed, re-estlmate, with FRM (Phase 1), the f (J.) 
\ of' the/ remaining item and* repeat stepa 1-8 until no items arhl ' 
removed . a 

> . 414 
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7> Estjjnatdon of Competency ^ ' 

We do riot consider the* estimation of the CjOmpetency parameter of the 

; ' ' ' ■ ■ ''''' - ■ ■ V .• 

BMM to be^ part of the FRM although^ it is a ^ part of COMET This arises 
because- it is usually desired to estimate comf>ete)icy on examination j 
responses resulting from administering tests composed of dnly^a few, 

not all, of the available items. Sinc§i at the tiinie FRM is utilized' . 

. " "... ■ • * . . ^ 

.these subsets have- not been selected ,^ tfife yC^) estimation process is 

' . ' » - • • / . ' ' - " ' ■ " ■ _ ' . .J^ 

delayed. ° - ;^ ' f 

Following Wright ajid Panchapakesan . (1969) , Moonan and Covher (1973) 

have programmed a computier'' subroutine calle| "Maxco" to estimate y(ri) 

\ from knpwlGidge of total-( scores^ and the da^iness of, items on an Sxamina- 

rion. This* process uses the Newtori-Raphson iteration method to esti- 

mate Y(n) corresppnding to :^aw score ranging, from 1 to k^l iu a test 

/composed of k^items. Y(n) for r^w scores CP^^S K are non-estimable,' 

t^e teg^t being either too diffj^cult ot 'too easy for the examinee. *We do 

knpw, * however , that the competencies are either less than the competency 

associated wi.th a raw^ score of ♦ 1 or greater tha^i the competency assb- 



ciat6d with a raw score of k. The formula iterated in'-Maxco is 

v/here r is a raw score on a test composed of k items whose easiness are 
estimated by.e(k) and y.X^), is the comptency of person n w^o^obtained a 
s raw score of r on the k items. 

We have used Maxco on the I = K = 7 items of the FRM numerical 
example giving the results in Exhibit 5. * . • . . 



\ 
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EXHIBIT 5 



Competency Estipi^tes for an^Exainination ) 
"* . of I « K « 7 ; ETST Items 



Raw' Sid^re 



' Competency Estimat4 













, non-estimalJle • 


'J 

n .-■ ■ 


•f ■ > ' ■ , 


\, .153 ..■ , 






.382 ■ 










> ; ; 4' ■ ■ . . 


"y'.- : a.363 . V 










. 6 

o 






7 . - 


noh-estimable 





The standard errors 4£ these estimates are not provided here. 



fit- 
i'.i 



a'-*- 



0 



. :/ 



416 



0 

ERIC 



8. Th^ COMET. System ^ " , ' ' \ 

The goal Cornet Us to provide an integrated; systematic and objec- 
tive procBf^lure f or the assessment of trainees in military programs which 
is based uppri-tbe Rasch- Binary Measurement Modal. : 

■ . \ . ■ . ' ' * ^ '\ ' ■ ' ■ 

The entire %stem is composed of 12 interjelated" sub-sy stems . and 
^he ohp=ectives of COMET ate to identify, ^ design 
' sub-systems andi integrate t^m Into an sy^^^e^t^ who 

. This Vaper' s_content :is' not intended to accomplish kll of ^that but 
. is intended as the' fir sj step; vis;.,* the project definition and concep- 
tual design, phase . Actually it has do^^^ ^ore;^ than that since the; 
deyelopment of the FRM, shbwA jhere, Was^ *: 
wK^ch the whale^^ system would nbt- be efficient or even f^^ible* We intend 
next t^o specify; the essential^^su describe their 

f unctiGnS. The following sub-systems have been >idi^ntif - 



A/ Subf System For 

a. training needs assessment 

b. transforming training needs 

inta learning ob j ec tives 



^Objectives, 





"etr^^ining course design . 




d. 


developing test items from : 
iearniiig Jdtgectives 




e, 
• 


" iteia response sampling : 


^ .-Samplye 




data, collection \ • 


■ ■•■;.D^ta^ - ■ 




data analysis 


FRM 




examination design 


Test 


i. 


test reproduction \ 


Repro 


j-" 


" competency estimation 
evaluation \ 


• Meas^ 


k. 


Eval 


1. 


control 


, Cont 



D escription ^of Sub-System Inunctions - ' ■ - 

Needs , . ^ 

■ f 

Planning the COMET system should start with the. identification 
of training needs. A training need is defined as the discrepency 
between desired and required outcomes. The important notion is 
that to have a "need" we must identif^^ and document* that there is 
a gap between two outcomes, that which is currently resulting and 
. ,that which s^^ould b6 resulting. . A. training specialist is pirobably 
required to design and operate this sub-system. 

■ ' . , ■ ■ . ■ ■ ' i '■- ^ . 

Objectitres "* 

" It is convenient and perhaps necessary to transform identified . 

needs into learning objective statements. There are several current 
;> means for doing this but none which the author cares to recommend. 
The training specialist should also be helpful in this sub-system, 

' : • •-. - ■ . ■■ ■ ■ ^ ■ ^ . .. 

Course 

As was previously suggested, a training course design procedure 

• ne^^o^be followed to implenfent this sub-system. Probably no 
bett^ source for thi,s effort has been prepared other than that 

* of Rundquist (1967). 

Items y ' • 

Having designed the training pburse and specified learning 
' objectives we need to prepare items which, if answered correctly, 
wiV- reflect .trainee competency. This process may be significantly 
shortened by utilizing items made available: from different* services 
suggested bjr Wood (1968). Moonan (1972) provides a table gf non- 
^ militailr behavioral objective and test item j^^k^ ^ 

' 418 r 



■ ■ ■ ' 

Sample * ' . ' 

* ■ •« . • .* • 

Havdng conott^uqt^d the Iteia^s for measuring competency we are 

... • • . . ^ ■ . - ; 

required next to administer t^»Bm Items, perhaps in sm^'ll iNitehes, 
to individuiO^ of vatyljotg tralnin* comp«^tency. As Uidi^tpA »atliJtt 
no formal sa^pi;tog plnn Is requUred but r^pon^es may \0 acquired 

on a so-called "catch as cat^jh can^* l>asis. , * 

♦ ■ ■ • ■ * 

Data 

these resppn^ data, may be rec6t4ed Ijx varlods ways such as 
port-a-punch or mark sense cards or ^>by an opftical acaAAer . form.. An 



optical scanner system together with its mini-tawuCrir appears tp 

. fly . . , i J . > d * 

be the optimal d^ta collection and processing sj^st^m for'COjlET. The 
costj effectiveness of this suggestion needs to he examined in any 
particular application., ^ .. - ■ ^ 

FBM > ^ . * . 

Uaving collected the response data^ it needs to be organized in 
SCi,j) fo™ antf analyzed witl^^ the frequency ratio method, FR>: in 

order to estimate the item easiaess and to test the goodness of fit 

■f • • . . ■ 

of the response data to the BMM. Items not 'fitting the model should 
he examined and purged from the item bank. 
Test and Reprb 

A subV^stem which canMesign automatically, ' prepare anc uupli- 
<cate compietency examinations from the available item pool is required 
This may be done via the computer or with the clerical system used by 
GLAK2S. This process is sometimes referred to as CA'iC: Computer 
Assisted Test Construction. 
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• Having specified the^pxamination and the items therein we 
•require the deyelopment of competency tables associated with pos- 
sible raw scores on the^e tests. This ^is accomplished by. using 
the Maxdo subrouj/ine developed by Moonan and^ Covher (1973). - > 
Eval ■ 

Having made the measurements of training competency we can 
use this information in a variety of ways to make evaluations 
among which are following: • ' 

a. Trainee competency \ 

b. ^ Change in traj^aee competency 

C» Instructor capabilities 

■ ■ - • i ' ■ " • 

J ^ d. Training facility effectiveness , 

f • • . ' '■' ' 

el Training program^ effectiveness 

' '/ ■ ' V- ' ' ' ' ' ' ■ r- ' ■ ' 

*f. Program objectively t 

A good discussion of these and other ways that the BMM can be 

I 

used in ev^aluation is given in Hiscdx (1974) • ' 

cost . . V . ■ .■ 

The COMET system needs to- be controlled as a systen. :A an 
effective manager who must constantly carry out the six functions 
of a manager: pXarm±6g, directing, organizing, staffing, evaluat- 
ing and controlling. 

Any operational COMET system needs to be developed through 
systems analysis by an experienced or professional systems analyst. 



9. Expected Implementation Problems 

" It is, expected that the most difficult problem to be encountered in 
implementing COMET would be to 'find and utilize a systems' analyst to 

'design the system for local conditions and constraints as well as to 
appoint a competent manager to monitor and^manage the aperational' sys- 
tem. This person must possess many skills and have knowledge in tlje . 
areas of measurement, computers, e'd'iication. training. . systems etc . as' ^ 
well as., the ability to' work with specialists in t^ose fields and with 

•.inilitary. personnel and systems. Such competent persons are difficult to 
locacel . . * * ... 

. . Developing or finding items for the competency, pool may be onerous^ but 
noc coo difficult. " Interservice cooperation should be sought 'here and' 
'tl^e llTA could .play' a significant and helpful' role in *these matters , A 
prototype COMET system might be designed and de^|l^4 wit^hin one ser- 
vice and developmental costs shared. This system mjght .hen be used by 
Other interested parties. 

' Standard- errors of estimates of v(n) and e(i);are. needed but not pro- ' 
.Ided in this report. Their specification is under development b.c was 
.oc completed in time for inclusion in this paper. The science of |. 
examination design needs better formulation so that the test sub-syste. 
may make use of it. There appears to be only crude rules of thumb anc 
theo^ available in psychometrics\to assist us in this area.. 

All in all the practical problems do not appear, too limiting. .what 
is most needed is the appearance of a dynamic, resourceful, and persua- 
' leader to step forth from '^ong our ranks and pursue the development 
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'and implementation of a COMET-like system within a training facility of 
his own service, . ^ . 

The^ajor conceptual design work has been shown here, but there are 
many details yet to be worked out. 

■■ ■ ' . s 
• ' ■ ■ I - ■ ' , .■ " o 

10, ^ Summary * 

This report proposed, defined the scope and outlined the 12 sub- 
systems of -'a general system called COMET whose general purpose -is co 
objectively assess the competencies attained by trainees during the 
course of training dn the military service. The measurement and evalua- 

t ' ■ ■ ^ . » 

tion features of COMET are based on a binary mea^surement model created^ 

■ ■ * 

by George Rasch* A new analysis procedure was developed tp estimate' a 

parameter of the model and tp test 'the goodness of fit of reSpfonses of 
■ . • fit 

test items to the model. A numeri(^al example of computations was pro- 

vided and practical proW.ems expected t;o be encountered during ituplemen-- 
tation are mentioned.' 
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